Image data processing method and image processing processor

ABSTRACT

A method of processing image data includes: receiving image data from a color filter array including N×N same color pixels; converting first pixel data in an N×N array into second pixel data in an (N−L)×(N−M) array, wherein the first pixel data is output from the N×N same color pixels; and generating third pixel data in the N×N array by performing reconstruction on the second pixel data, wherein each of “L” and “M” is a natural number that is greater than or equal to 1 and less than N, and “N” is a natural number that is greater than or equal to 2.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. § 119 to Korean PatentApplication No. 10-2022-0059631, filed on May 16, 2022, in the KoreanIntellectual Property Office, the disclosure of which is incorporated byreference herein in its entirety.

TECHNICAL FIELD

The present inventive concept relates to an image data processing methodand an image processing processor.

DISCUSSION OF THE RELATED ART

Generally, a color filter including red, green, or blue may be above animage sensor for the color reproduction of an image, and may be arrangedin the form of an array. The array of color filters may be referred toas a color filter array (CFA). In addition, a single microlens may bedisposed above a CFA such that the microlens is across several pixels ofthe same color filter.

SUMMARY

The present inventive concept provides an image data processing method,which is applicable regardless of the cause of a phase artifact and bywhich a loss of resolution is reduced by making full use of pixelinformation of the same color.

According to an aspect of the present inventive concept, there isprovided a method of processing image data output from a color filterarray including N×N same color pixels.

According to an embodiment of the present inventive concept, a method ofprocessing image data includes: receiving image data from a color filterarray including N×N same color pixels; converting first pixel data in anN×N array into second pixel data in an (N−L)×(N−M) array, wherein thefirst pixel data is output from the N×N same color pixels; andgenerating third pixel data in the N×N array by performingreconstruction on the second pixel data, wherein each of “L” and “M” isa natural number that is greater than or equal to 1 and less than N, and“N” is a natural number that is greater than or equal to 2.

According to an embodiment of the present inventive concept, a method ofprocessing image data includes: receiving image data output from a colorfilter array (CFA) including a plurality of CFA blocks including atleast one color region including pixels of a same color; generatingsecond pixel data by converting first pixel data output from the atleast one color region; and generating third pixel data by performingreconstruction on the second pixel data, wherein the generating of thesecond pixel data includes forming a plurality of nodes includinginformation about pixels included in the first pixel data.

According to an embodiment of the present inventive concept, an imageprocessing processor processing image data output from an image sensorincludes: a first processing circuit configured to perform conversion ofthe image data received from the image sensor; and a second processingcircuit configured to perform reconstruction on converted data, whereinthe first processing circuit is configured to perform the conversion bybinning the image data including pixel data in an N×N array into pixeldata in an (N−L)×(N−M) array, where each of “L” and “M” is a naturalnumber that is greater than or equal to 1 and less than “N”, “N” is anatural number that is greater than or equal to 2, and the secondprocessing circuit is configured to perform the reconstruction byinterpolating nodes corresponding to the pixel data in the (N−L)×(N−M)array.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects of the presentative concept will become moreapparent by describing in detail embodiments thereof, with reference tothe accompanying drawings, in which:

FIG. 1A illustrates an example of a color filter according to anembodiment of the present inventive concept;

FIG. 1B is a diagram illustrating the brightness when a light sourceilluminates the color filter of FIG. 1A according to an embodiment ofthe present inventive concept;

FIG. 2 is a block diagram of an image processing device according to anembodiment of the present inventive concept;

FIGS. 3A, 3B, and 3C are diagrams of pixel arrays according to anembodiment of the present inventive concept;

FIG. 4 is a diagram for describing a processing method performed by animage processing device, according to an embodiment of the presentinventive concept;

FIGS. 5A, 5B, and 5C are diagrams for describing conversion methodsperformed by an image processing device, according to an embodiment ofthe present inventive concept;

FIGS. 6A, 6B, 6C, 6D, and 6E are diagrams illustrating convertingresults of an image processing device, according to an embodiment of thepresent inventive concept;

FIG. 7A illustrates nodes resulting from conversion, according to anembodiment of the present inventive concept;

FIG. 7B illustrates reconstructed pixel data according to an embodimentof the present inventive concept;

FIG. 7C is a diagram for describing the relationship between FIG. 7A andFIG. 7B;

FIGS. 8, 9, and 10 are diagrams for describing reconstruction methodsaccording to an embodiment of the present inventive concept; and

FIGS. 11A and 11B are flowcharts of an image data processing methodaccording to an embodiment of the present inventive concept.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, various embodiments of the present inventive concept aredescribed with reference to the accompanying drawings.

FIG. 1A illustrates an example of a color filter according to anembodiment of the present inventive concept. FIG. 1B is a diagramillustrating the brightness when a light source illuminates the colorfilter of FIG. 1A according to an embodiment of the present inventiveconcept.

Referring to FIG. 1A, a color filter 10 of a Bayer pattern includingcolor pixels in a 4×4 array is provided. Each of first to fourth colorpixels 11, 12, 13, and 14 in the 4×4 array may include pixels having thesame color. Referring to FIG. 1A, a plurality of microlenses 20 may berespectively disposed on the first to fourth color pixels 11, 12, 13,and 14 in the 4×4 array. Referring to FIG. 1A, four microlenses 20 maybe disposed on the first color pixels 11. Referring to FIG. 1A, fourmicrolenses 20 may be respectively disposed on the second color pixels12, four microlenses 20 may be respectively disposed on the third colorpixels 13, and four microlenses 20 may be respectively disposed on thefourth color pixels 14. Each of the microlenses 20 may have a sizecorresponding to 2×2 array color pixels.

FIG. 1B illustrates an example, in which there is a difference in apixel value of the same color in a cell, in which 2×2 pixels share asingle microlens with one another, according to a light illuminationdirection. Light from a light source L may pass through a microlens 20and a color pixel CP, and image data 141 may be generated. Referring toFIG. 1B, it may be seen that there is a difference in the brightness ofone color pixel CP according to the position of the light source L.

In the case of an image sensor including a microlens, the path of lightincident to the microlens may vary with the position of a light sourceL, based on the optical characteristics of the image sensor. In otherwords, in the case of an image sensor including a microlens 20, anintra-channel difference may occur according to a phase in the samecolor channel because of various causes, such as a structure issue, alimit in processes, a position or intensity of a light source, and focusbehavior.

FIG. 2 is a block diagram of an image processing device according to anembodiment of the present inventive concept.

An image processing device 100 may include an electronic device thatcaptures and displays an image or performs an operation based on acaptured image. For example, the image processing device 100 may includea personal computer (PC), an Internet of things (IoT) device, or aportable electronic device. The portable electronic device may include,for example, a laptop computer, a mobile phone, a smartphone, a tabletPC, a personal digital assistant (PDA), an enterprise digital assistant(EDA), a digital still camera, a digital video camera, an audio device,a portable multimedia player (PMP), a personal navigation device (PND),an MP3 player, a handheld game console, an e-book, or a wearable device.The image processing device 100 may be mounted on an electronic device,such as a drone or an advanced drivers assistance system (ADAS), or onan electronic device provided as a component of a vehicle, furniture, amanufacturing facility, a door, or various kinds of measuring equipment.

Referring to FIG. 2 , the image processing device 100 may include animage sensor 110 and an image processing processor 130. The imageprocessing device 100 may further include other elements, such as adisplay and a user interface.

The image sensor 110 may include a pixel array, which includes aplurality of pixels arranged in two dimensions, and a readout circuit.The pixel array may receive and convert optical signals into electricalsignals. For example, the pixel array may include a photoelectricconversion element, such as a charge-coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS) conversion element, orother various kinds of photoelectric conversion elements. The readoutcircuit may generate raw data, based on an electrical signal from apixel array, and may output the raw data, which has undergone noiseremoval or the like, as first image data IDATA. The image sensor 110 maybe implemented in a semiconductor chip or a package, which includes thepixel array and the readout circuit.

According to an embodiment of the present inventive concept, the imagesensor 110 may include a pixel array, and a color filter array (CFA) 120may be disposed on the pixel array such that a certain color componentis sensed by each pixel. The image sensor 110 may include the CFA 120having a certain pattern, and may convert an optical signal of an objectincident through an optical lens LS into an electrical signal using theCFA 120. In addition, the image sensor 110 may generate and output thefirst image data IDATA based on the electrical signal. The optical lensLS may include a microlens. There may be a plurality of optical lensesLS. Although it is illustrated in FIG. 2 that the optical lens LS isseparate from the image sensor 110, the optical lens LS may be includedin the image sensor 110.

In the description of embodiments below, the terms “color filter”,“color pixel”, “filter array”, and “pixel array” may be variouslydefined. For example, a CFA may be defined as a separate element, whichis disposed on a pixel array including a photosensitive device, or asbeing included in a pixel array. A color pixel may be defined asincluding a color filter corresponding thereto. A CFA cell and a CFAblock may each be defined as including the color pixel described above.

The CFA 120 may include a plurality of CFA blocks repeatedly arranged inhorizontal and vertical directions. Each CFA block may include colorpixels having a certain size. According to an embodiment of the presentinventive concept, the CFA blocks may be arranged in a pattern.According to an embodiment of the present inventive concept, the CFAblocks may be arranged in a Bayer pattern. According to an embodiment ofthe present inventive concept, the CFA blocks may be arranged in variouspatterns, such as a tetra pattern in which a single color unit has asize of 2×2, a nona pattern in which a single color unit has a size of3×3, and a hexadeca pattern in which a single color unit has a size of4×4. For convenience of description, it is assumed that the CFA blocksof the CFA 120 are arranged in a Bayer pattern. However, the presentinventive concept is not limited thereto.

The image processing processor 130 may include a first processingcircuit 131, a second processing circuit 132, and a memory 133. Thefirst processing circuit 131 and the second processing circuit 132 maybe implemented in one or more semiconductor chips. For example, theimage processing processor 130 or the image processing device 100 may beimplemented in a system-on-chip (SoC).

The image processing processor 130 may convert the format of the firstimage data IDATA received from the image sensor 110 by performing aprocessing operation including remosaicing and demosaicing on the firstimage data IDATA.

The first processing circuit 131 of the image processing processor 130may perform conversion on the first image data IDATA. The firstprocessing circuit 131 may perform conversion on the first image dataIDATA by binning the first image data IDATA including N×N pixel datainto (N−L)×(N−M) pixel data. At this time, L and M may each be a naturalnumber that is greater than or equal to 1 and less than N. N may be anatural number that is greater than or equal to 2. The conversion methodperformed by the first processing circuit 131 is described in detailwith reference to FIGS. 5A to 6E below.

The second processing circuit 132 of the image processing processor 130may perform reconstruction on image data IDATA′ resulting from theconversion by the first processing circuit 131. The second processingcircuit 132 may perform reconstruction of data by interpolating a nodecorresponding to each piece of (N−L)×(N−M) pixel data. According to anembodiment of the present inventive concept, to reconstruct pixel datasurrounded by four nodes among the nodes respectively corresponding tothe pieces of (N−L)×(N−M) pixel data, the second processing circuit 132may perform interpolation from an average value of the four nodes.According to an embodiment of the present inventive concept, toreconstruct the data of a pixel contacting one or two nodes among thenodes respectively corresponding to the pieces of (N−L)×(N−M) pixeldata, the second processing circuit 132 may perform interpolation on theone or two nodes by using linear interpolation or second-orderpolynomial interpolation. The interpolation method performed by thesecond processing circuit 132 is described in detail with reference toFIGS. 7A to 10 below.

Reconstructed image data IDATA″ of the second processing circuit 132 maybe stored in the memory 133.

When an image is processed by the image processing device 100, accordingto an embodiment of the present inventive concept, a phase artifact inthe first image data IDATA may be removed.

FIGS. 3A to 3C are diagrams illustrating examples of a pixel arraycorresponding to the CFA 120 in FIG. 2 .

Referring to FIG. 3A, a pixel array PX_ARRAY may include a plurality ofpixels in a plurality of rows and columns. For example, a shared pixel,which is defined as a unit including pixels in two rows and two columns,may include four sub-pixels. In other words, the shared pixel mayinclude four photodiodes respectively corresponding to four sub-pixels.The pixel array PX_ARRAY may include first to sixteenth shared pixelsSP0 to SP15. The pixel array PX_ARRAY may include color filters suchthat the first to sixteenth shared pixels SP0 to SP15 may sense variouscolors. For example, the color filters may include a filter sensing ared color R, a filter sensing a green color G, and a filter sensing ablue color B. Each of the first to sixteenth shared pixels SP0 to SP15may include sub-pixels provided with the same color filter. For example,each of the first shared pixel SP0, the third shared pixel SP2, theninth shared pixel SP8, and the eleventh shared pixel SP10 may includesub-pixels provided with a blue color filter. Each of the second sharedpixel SP1, the fourth shared pixel SP3, the fifth shared pixel SP4, theseventh shared pixel SP6, the tenth shared pixel SP9, the twelfth sharedpixel SP11, the thirteenth shared pixel SP12, and the fifteenth sharedpixel SP14 may include sub-pixels provided with a green color filter.Each of the sixth shared pixel SP5, the eighth shared pixel SP7, thefourteenth shared pixel SP13, and the sixteenth shared pixel SP15 mayinclude sub-pixels provided with a red color filter. A group includingthe first shared pixel SP0, the second shared pixel SP1, the fifthshared pixel SP4, and the sixth shared pixel SP5, a group including thethird shared pixel SP2, the fourth shared pixel SP3, the seventh sharedpixel SP6, and the eighth shared pixel SP7, a group including the ninthshared pixel SP8, the tenth shared pixel SP9, the thirteenth sharedpixel SP12, and the fourteenth shared pixel SP13, and a group includingthe eleventh shared pixel SP10, the twelfth shared pixel SP11, thefifteenth shared pixel SP14, and the sixteenth shared pixel SP15 mayeach be disposed on the pixel array PX_ARRAY in a Bayer pattern.According to an embodiment of the present inventive concept, the groupincluding the first shared pixel SP0, the second shared pixel SP1, thefifth shared pixel SP4, and the sixth shared pixel SP5, the groupincluding the third shared pixel SP2, the fourth shared pixel SP3, theseventh shared pixel SP6, and the eighth shared pixel SP7, the groupincluding the ninth shared pixel SP8, the tenth shared pixel SP9, thethirteenth shared pixel SP12, and the fourteenth shared pixel SP13, andthe group including the eleventh shared pixel SP10, the twelfth sharedpixel SP11, the fifteenth shared pixel SP14, and the sixteenth sharedpixel SP15 may each correspond to a CFA block.

However, this is just an example, and the pixel array PX_ARRAY mayinclude various kinds of color filters, according to embodiments of thepresent inventive concept. For example, color filters may include afilter sensing a yellow color, a filter sensing a cyan color, a filtersensing a magenta color, and filter sensing a green color. In addition,color filters may include a filter sensing a red color, a filter sensinga green color, a filter sensing a blue color, and a filter sensing awhite color. The pixel array PX_ARRAY may include more shared pixels,and the first to sixteenth shared pixels SP0 to SP15 may be variouslyarranged.

Referring to FIG. 3B, each of the first shared pixel SP0, the secondshared pixel SP1, the fifth shared pixel SP4, and the sixth shared pixelSP5 may include nine sub-pixels. The first shared pixel SP0 may includenine sub-pixels provided with a blue color filter, and each of thesecond and fifth shared pixels SP1 and SP4 may include nine sub-pixelsprovided with a green color filter. The sixth shared pixel SP5 mayinclude nine sub-pixels provided with a red color filter. In someembodiments of the present inventive concept, each of the first, second,fifth, and sixth shared pixels SP0, SP1, SP4, and SP5 may be referred toas a nova cell.

Referring to FIG. 3C, each of the first, second, fifth, and sixth sharedpixels SP0, SP1, SP4, and SP5 may include sixteen sub-pixels. The firstshared pixel SP0 may include sixteen sub-pixels provided with a bluecolor filter, and each of the second and fifth shared pixels SP1 and SP4may include sixteen sub-pixels provided with a green color filter. Thesixth shared pixel SP5 may include sixteen sub-pixels provided with ared color filter. In some embodiments of the present inventive concept,each of the first, second, fifth, and sixth shared pixels SP0, SP1, SP4,and SP5 may be referred to as a hexadeca cell.

Hereinafter, a shared pixel includes N×N sub-pixels, and a method ofprocessing pixel data of N×N sub-pixels is described.

FIG. 4 is a diagram for describing an image data processing methodaccording to an embodiment of the present inventive concept.

FIG. 4 illustrates an example of image data of four shared pixels 1211,1212, 1213, and 1214 each including N×N sub-pixels.

The first picture in FIG. 4 may refer to first image data 1210 outputfrom a CFA having a Bayer pattern. The second picture in FIG. 4 mayrefer to second image data 1220, which includes (N−1)×(N−1) sub-pixelsgenerated through conversion of the first image data 1210. The thirdpicture in FIG. 4 may refer to third image data 1230, which includes N×Nsub-pixels generated by performing reconstruction on the second imagedata 1220. The first image data 1210 in FIG. 4 may be referred to asfirst pixel data 1210. The second image data 1220 in FIG. 4 may bereferred to as second pixel data 1220. The third image data 1230 in FIG.4 may be referred to as third pixel data 1230. The term “pixel data”used herein may refer to image data that includes informationcorresponding to a pixel and has a form of a pixel array.

FIG. 4 illustrates a method of processing the first pixel data 1220including N×N sub-pixels. FIG. 4 illustrates an example of convertingthe first pixel data 1210 including N×N sub-pixels into the second pixeldata including (N−1)×(N−1) sub-pixels and reconstructing the third pixeldata 1230 including N×N sub-pixels from the second pixel data 1220including (N−1)×(N−1). According to an embodiment of the presentinventive concept, N is a natural number that is greater than or equalto 2.

According to an embodiment of the present inventive concept, aconversion method may include binning. According to an embodiment of thepresent inventive concept, a microlens 1211 a may be disposed on aplurality of sub-pixels included in the shared pixel 1211 as shown inFIG. 4 . FIG. 4 illustrates an example, in which four sub-pixelsincluded in the shared pixel 1211 may share one microlens, e.g., themicrolens 1211 a, with one another. According to the present inventiveconcept, to reduce a phase difference that occurs when one microlens isshared by several pixels in a CFA block (e.g., the shared pixel 1211,1212, 1213, or 1214) including the same color pixels, phase informationof adjacent sub-pixels may be used. For this operation, an averageoperation on pixel data values in a 2×2 unit may be performed in ashared pixel including the same color region, with a pixel-by-pixelshift. The average operation is described in detail with reference toFIGS. 5A to 6E.

According to an embodiment of the present inventive concept, the secondpixel data 1220 may include (N−1)×(N−1) pieces of pixel data. The secondpixel data 1220 may include (N−1)×(N−1) nodes. According to anembodiment of the present inventive concept, the third pixel data 1230may be reconstructed from the second pixel data 1220 by performinginterpolation on the second pixel data 1220. The third pixel data 1230may be reconstructed using linear interpolation on some of the nodes ofthe second pixel data 1220 or interpolation using a second-orderpolynomial of some nodes. This is described in detail with reference toFIGS. 7A to 10 .

Referring to FIG. 4 , the first pixel data 1210 may include phaseinformation. The second pixel data 1220 resulting from conversion mightnot include the phase information. The third pixel data 1230 resultingfrom reconstruction might not include the phase information and may havethe same resolution as the first pixel data 1210. In other words, thephase information included in the first pixel data 1210 may be removedduring the conversion.

According to an embodiment of the present inventive concept, image datamay be processed by a method of down-sampling N×N sub-pixel data to(N−1)×(N−1) sub-pixel data, and then up-sampling the (N−1)×(N−1)sub-pixel data to N×N sub-pixel data. According to an embodiment of thepresent inventive concept, regardless of the occurrence pattern, i.e.,periodicity or aperiodicity, of a phase artifact, data may be convertedthrough down-sampling, with the full use of pixel information of thesame color in a spatial domain. In addition, image data with a reducedloss of resolution may be generated through up-sampling.

Although data processing is performed using one-step down-sampling andone-step up-sampling in an embodiment of the present inventive concept,the present inventive concept is not limited thereto. According to anembodiment of the present inventive concept, N×N pixel data may beconverted into (N+1)×(N+1) pixel data, and N×N pixel data may bereconstructed from the (N+1)×(N+1) pixel data.

FIGS. 5A to 5C are diagrams for describing conversion methods accordingto an embodiment of the present inventive concept.

FIG. 5A illustrates an example of converting pixel data including 4×4sub-pixels into sub-pixel data including 3×3 sub-pixels. The firstpicture in FIG. 5A may illustrate an example of the pixel data including4×4 sub-pixels. Referring to the first picture in FIG. 5A, the microlens1211 a may be provided in a region corresponding to 2×2 sub-pixels in4×4 sub-pixel data. Referring to the first and second pictures in FIG.5A, a region corresponding to the microlens 1211 a in a 2×2 array may bea unit of conversion. According to an embodiment of the presentinventive concept, conversion may be performed while a unit ofconversion is sequentially shifted across 4×4 sub-pixels. According toan embodiment of the present inventive concept, a conversion method mayinclude binning.

Referring back to FIG. 5A, while sliding an A region corresponding to amicrolens in a 2>2 array, a node N may be generated by calculating anaverage value of data in the A region. According to an embodiment of thepresent inventive concept, the node N may be generated by calculating anaverage value of pixels corresponding to a unit region, on which aconversion is performed. The node N generated as a result of theconversion may include average value data of pixels surrounding the nodeN. The node N may include data information of four adjacent pixels. As aresult of the conversion, a 4×4 color pattern may be converted into a3×3 color pattern, and simultaneously, artifacts caused by a phasedifference in each pixel may be removed.

FIG. 5B illustrates an example of converting pixels in a 6×6 array intopixels in a 4×4 array. Referring to FIG. 5B, a region corresponding to amicrolens 1211 b in a 3×3 array may be a unit of conversion. Referringto FIG. 5B, the unit of conversion may be a microlens in a 3×3 array,differently from FIG. 5A. Referring to FIG. 5B, while sliding an A′region corresponding to the microlens in a 3×3 array, a node N′ may begenerated by calculating an average value of data in the A′ region.

According to an embodiment of the present inventive concept, a unit ofconversion may be a region in which a microlens corresponds to a pixelarray. In other words, the unit of conversion may vary with the size ofa microlens. When the unit of conversion changes, the number of nodesgenerated through the conversion may also change. For convenience ofdescription, it is assumed that a unit of conversion is a regioncorresponding to a 2×2 microlens.

FIG. 5C is a diagram for describing an example of converting pixel dataincluding 4×4 sub-pixels. FIG. 5C illustrates a process of forming eachof the nodes by sequentially sliding a conversion unit A, whichcorresponds to a 2×2 microlens as shown in FIG. 5A, across 4×4sub-pixels. According to an embodiment of the present inventive concept,conversion may be performed by performing binning on all sub-pixels toinclude information of all sub-pixels.

According to an embodiment of the present inventive concept, informationon all phases may be reflected by performing an average operation onpixel values in an I×I unit with respect to sub-pixels in a color regionincluding pixels of the same color. At this time, the I×I unit maycorrespond to a region occupied by a microlens. The I×I unit may be aconversion unit. “I” may be a natural number that is greater than orequal to 2. Accordingly, the size of an image may be compressed from N×Nto (N−L)×(N−M). According to the present embodiment of FIG. 5C, “I” maybe equal to 2.

Referring to FIG. 5C, one node may be formed per one conversion unit Ain a 2×2 array. According to an embodiment of the present inventiveconcept, an average value of four sub-pixels corresponding to aconversion unit in a 2×2 array may be set as a node value. Accordingly,when conversion is performed on pixel data including 4×4 sub-pixels byusing a 2×2 microlens as a conversion unit, a total of 3×3 pieces ofsecond pixel data may be generated. The second pixel data may include3×3 nodes. According to an embodiment of the present inventive concept,the term “node” used herein may be interchangeably used with a “binningpoint”.

According to an embodiment of the present inventive concept, convertingpixel data in an N×N array into pixel data in an (N−1)×(N−1) array isprovided for convenience of description, but the present inventiveconcept is not limited thereto. In an embodiment of the presentinventive concept, pixel data in an N×N array may be converted intopixel data in an (N−2)×(N−2) array. In an embodiment of the presentinventive concept, pixel data in an N×N array may be converted intopixel data in an (N−1)×(N−2) array. In an embodiment of the presentinventive concept, pixel data in an N×N array may be converted intopixel data in an (N−L)×(N−M) array. At this time, “L” and “M” may eachbe a natural number that is greater than or equal to I and less than“N”. “N” may be a natural number that is greater than or equal to 2.Binning may be performed in other various manners than those describedabove and may be applied to various CFA patterns including thosedescribed above.

FIGS. 6A to 6E are diagrams illustrating various examples of secondpixel data resulting from conversion.

FIG. 6A illustrates an example of a result of performing a binningoperation on all available sub-pixels, as shown in FIGS. 5A to 5C.Referring to FIG. 6A, it may be seen that 3×3 nodes are formed as aresult of performing a binning operation on 4×4 sub-pixels.

Referring to FIGS. 6B to 6E, a binning operation may be sparselyperformed for the efficiency thereof.

FIG. 6B illustrates an example of a result of performing a binningoperation with five binning points being set. According to an embodimentof the present inventive concept with reference to FIG. 6B, the binningoperation may be performed to output a node at the center of a grid of4×4 sub-pixels and to output nodes surrounding the central node in anX-axis direction and a Y-axis direction, which are perpendicular to eachother. For example, four nodes may surround the central node; however,the present inventive concept is not limited thereto.

FIG. 6C illustrates an example of a result of performing a binningoperation with four binning points being set. According to an embodimentof the present inventive concept with reference to FIG. 6C, the binningoperation may be performed to output the nodes in the embodiment of FIG.6B, except for the central node. For example, the nodes may surround acentral area of a grid of 4×4 sub-pixels.

FIG. 6D illustrates an example of a result of performing a binningoperation with five binning points being set. According to an embodimentof the present inventive concept with reference to FIG. 6D, the binningoperation may be performed to output a node at the center of a grid of4×4 sub-pixels and output nodes adjacent to the central node in diagonaldirections of the central node.

FIG. 6E illustrates an example of a result of performing a binningoperation with four binning points being set. According to an embodimentof the present inventive concept with reference to FIG. 6E, the binningoperation may be performed to output the nodes in the embodiment of FIG.6D, except for the central node at the center of a grid of 4×4sub-pixels. For example, nodes may be output to be adjacent to a centralarea of the grid of 4×4 sub-pixels in a diagonal direction of thecentral area. For example, the diagonal direction may be with respect tothe X-axis direction and the Y-axis direction, and may cross the X-axisdirection and the Y-axis direction.

According to an embodiment of the present inventive concept, the binningmethods of FIGS. 6B to 6E may be used when conversion is desired to beperformed on some sub-pixels having a large difference from the valuesof the other sub-pixels.

Hereinafter, a method of reconstructing data is described, based on theassumption that binning is performed on all sub-pixels as shown in FIG.6A. The interpolation method described below may also be applied to aninterpolation method for the case of performing sparse conversion asshown in FIGS. 6B to 6E.

A phase artifact may be removed as a result of binning N×N sub-pixeldata into (N−1)×(N−1) sub-pixel data. However, because the size of animage is reduced by converting the N×N sub-pixel data into the(N−1)×(N−1) sub-pixel data, the resolution of the image may decrease.Therefore, the N×N sub-pixel data may be reconstructed from the(N−1)×(N−1) sub-pixel data. The reconstruction method is described indetail with reference to FIGS. 7A to 7C below.

FIG. 7A illustrates an example of a plurality of nodes generated throughconversion. FIG. 7B illustrates an example of final pixel data generatedthrough reconstruction. FIG. 7C is a diagram for describing therelationship between FIG. 7A and FIG. 7B.

FIG. 7A illustrates a plurality of nodes N1 to N9 in a 3×3 array.According to an embodiment of the present inventive concept, each of thenodes N1 to N9 in FIG. 7A may include first pixel data adjacent thereto.

FIG. 7A illustrates 3×3 nodes, i.e., the nodes N1 to N9, resulting fromconverting 4×4 pieces of sub-pixel data. FIG. 7A illustrates 4×4 piecesof pixel data PD1 to PD16 included in first pixel data together with thenodes N1 to N9 resulting from the conversion on the first pixel data.Referring to FIG. 7A, it may be seen that information of pixels of thefirst pixel data, which surround each node, is binned into the node.According to an embodiment of the present inventive concept, the node N1may include the pixel data PD1, PD2, PD5, and PD6.

FIG. 7B illustrates third pixel data, which includes pixel data PD1′ toPD16′ in a 4×4 array and is reconstructed using the nodes N1 to N9 inFIG. 7A. The third pixel data in a 4×4 array may be divided into threekinds of pixel data. According to an embodiment of the present inventiveconcept, the third pixel data may include central pixel data, outerpixel data, and corner pixel data. According to an embodiment of thepresent inventive concept, the outer pixel data and the corner pixeldata may be edge pixel data. Referring to FIG. 7B, there may be eightpieces of outer pixel data, four pieces of corner pixel data, and fourpieces of central pixel data. According to an embodiment of the presentinventive concept, the first pixel data may have the same array as thethird pixel data.

FIG. 7C illustrates the relationship between the nodes N1 to N9resulting from conversion and the pixel data PD1′ to PD16′ of the thirdpixel data resulting from reconstruction. According to an embodiment ofthe present inventive concept, each piece of the central pixel data maybe surrounded by four nodes. According to an embodiment of the presentinventive concept, each piece of the outer pixel data may be adjacent totwo nodes. According to an embodiment of the present inventive concept,each piece of the corner pixel data may be adjacent to one node. Thethird pixel data resulting from the reconstruction may be divided intothe central pixel data, the outer pixel data, and the corner pixel dataaccording to the number of adjacent nodes resulting from binning.

An interpolation method used for each of central pixel data, outer pixeldata, and corner pixel data is described in detail below. According toan embodiment of the present inventive concept, the cases of cornerpixels, outer pixels, and central pixels are separately describedaccording to the positions thereof in an interpolated 4×4 array when 4×4information is reconstructed from compressed 3×3 information resultingfrom a binning operation.

FIG. 8 is a diagram for describing an interpolation method according toan embodiment of the present inventive concept.

Referring to FIG. 8 , central pixel data may be reconstructed bybilinear interpolation using four nodes surrounding the central pixeldata. This may be given by

${O_{target} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}I_{i}}}},$

where O_(target) may be central pixel data to be reconstructed, N may bethe number of nodes for the reconstruction, and “I” may be a data valueof the nodes for the reconstruction. Because the values of four nodessurrounding central pixel data reconstruct the central pixel data inFIG. 8 , “N” in the equation may be 4.

Referring to FIG. 8 , central pixel data PD6′ may correspond to anaverage of data values of the nodes N1, N2, N4, and N5 surrounding thecentral pixel data PD6′.

Referring to FIG. 8 , corner pixel data may correspond to the value ofone node adjacent to the coiner pixel data. According to an embodiment,corner pixel data may have a value of a node most adjacent thereto in adiagonal direction, based on nearest interpolation. This may be given by

O_(target)=I_(input),

where O_(target) may be corner pixel data to be reconstructed, andI_(input) may be a data value of one node adjacent to the corner pixeldata.

Referring to FIG. 8 , corner pixel data PD1′ may correspond to the datavalue of the node N1 nearest to the corner pixel data PD1′ in a diagonaldirection.

Referring to FIG. 8 , outer pixel data may correspond to an averagevalue of two nodes adjacent to the outer pixel data. According to anembodiment of the present inventive concept, outer pixel data may bereconstructed by linear interpolation on two nodes nearest to the outerpixel data. This may be given by

${O_{target} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}I_{i}}}},$

where O_(target) may be outer pixel data to be reconstructed, “N” may bethe number of nodes for the reconstruction, and “I” may be a data valueof the nodes for the reconstruction. Because the values of two nodesadjacent to outer pixel data reconstruct the outer pixel data in FIG. 8, “N” in the equation may be 2.

Referring to FIG. 8 , outer pixel data PD2′ may correspond to an averageof data values of the nodes N1 and N2 nearest to the outer pixel dataPD2′.

Because the number of nodes adjacent to corner pixel data and the numberof nodes adjacent to outer pixel data are limited, the corner pixel dataand the outer pixel data may be estimated using various interpolationmethods.

FIG. 9 is a diagram for describing an interpolation method according toan embodiment of the present inventive concept.

Redundant descriptions given above with reference to FIG. 8 may beomitted.

Referring to FIG. 9 , corner pixel data may be reconstructed byquadratic interpolation using a second-order polynomial using threenodes nearest to the corner pixel data in a diagonal direction. Forexample, the three nearest nodes may include the node nearest to thecorner pixel data, a central node at the center of a grid of 4×4sub-pixels, and a node nearest to another corner pixel data opposing thecorner pixel data. This may be given by

$\begin{matrix}{O_{target} = {{A \times I_{input}^{2}} + {B \times I_{input}} + C}} \\{\begin{bmatrix}A \\B \\C\end{bmatrix} = {\left( \begin{bmatrix}x_{0}^{2} & x_{0} & 1 \\x_{1}^{2} & x_{1} & 1 \\x_{2}^{2} & x_{2} & 1\end{bmatrix} \right)^{- 1}\begin{bmatrix}y_{0} \\y_{1} \\y_{2}\end{bmatrix}}}\end{matrix},$

where O_(target) may be corner pixel data to be reconstructed, x₀, x₁,and x₂ may each be a position value of a node undergoing interpolation,and y₀, y₁, and y₂ may respectively be input values respectivelycorresponding to x₀, x₁, and x₂. A, B, and C may be constants determinedby the above equation, and I_(input) may be a data value of a nodeadjacent to the corner pixel data.

Referring to FIG. 9 , the corner pixel data PD1′ may be estimated usingthe position values and input values of the nodes N1, N5, and N9 nearestto the corner pixel data PD1′ in a diagonal direction.

Referring to FIG. 9 , outer pixel data may be reconstructed by averagingpixel values estimated using a second-order polynomial in a horizontalor vertical direction based on a total of six nodes including two nodesnearest to the outer pixel data, wherein the six nodes are nearest tothe outer pixel data in the horizontal or vertical direction. This maybe given by

$\begin{matrix}{O_{target} = {{A \times I_{input}^{2}} + {B \times I_{input}} + C}} \\{\begin{bmatrix}A \\B \\C\end{bmatrix} = {\left( \begin{bmatrix}x_{0}^{2} & x_{0} & 1 \\x_{1}^{2} & x_{1} & 1 \\x_{2}^{2} & x_{2} & 1\end{bmatrix} \right)^{- 1}\begin{bmatrix}y_{0} \\y_{1} \\y_{2}\end{bmatrix}}}\end{matrix}.$

The same features of the equation as those given above may be omitted.Referring to FIG. 9 , the outer pixel data PD2′ may be estimated usingthe two nodes N1 and N2 nearest to the outer pixel data P12′ and thenodes N4, N7, N5, and N8 nearest to the outer pixel data P12′ in thevertical direction. For example, nodes N4 and N5 are closer to the outerpixel data PD2′ when compared to nodes N7 and N8.

According to an embodiment of the present inventive concept, the outerpixel data PD2′ may be estimated by averaging a result of interpolationusing a second-order polynomial of the nodes N1, N4, and N7 nearest tothe outer pixel data PD2′ in the vertical direction and a result ofinterpolation using a second-order polynomial of the nodes N2, N5, andN8 nearest to the outer pixel data PD2′ in the vertical direction.

FIG. 10 is a diagram for describing an interpolation method according toan embodiment of the present inventive concept.

Redundant descriptions given above with reference to FIG. 8 may beomitted.

Referring to FIG. 10 , corner pixel data may be reconstructed byquadratic interpolation using two nodes nearest to the corner pixel datain a diagonal direction. This may be given by

$\begin{matrix}{O_{target} = {{D \times I_{input}} + E}} \\{\begin{bmatrix}D \\E\end{bmatrix} = {\left( \begin{bmatrix}x_{0} & 1 \\x_{1} & 1\end{bmatrix} \right)^{- 1}\begin{bmatrix}y_{0} \\y_{1}\end{bmatrix}}}\end{matrix}$

where O_(target) may be corner pixel data to be reconstructed, x₀ and x₁may each be a position value of a node undergoing interpolation, and y₀and y₁ may respectively be input values respectively corresponding to x₀and x₁. D and E may be constants determined by the above equation, andI_(input) may be a data value of a node adjacent to the corner pixeldata.

Referring to FIG. 10 , the corner pixel data PD1′ may be estimated usingthe position values and input values of the two nodes N1 and N5 nearestto the corner pixel data PD1′ in a diagonal direction.

Referring to FIG. 10 , outer pixel data may be reconstructed using atotal of four nodes including two nodes nearest to the outer pixel data,wherein the four nodes are nearest to the outer pixel data in thehorizontal or vertical direction. This may be given by

$\begin{matrix}{O_{target} = {{D \times I_{input}} + E}} \\{\begin{bmatrix}D \\E\end{bmatrix} = {\left( \begin{bmatrix}x_{0} & 1 \\x_{1} & 1\end{bmatrix} \right)^{- 1}\begin{bmatrix}y_{0} \\y_{1}\end{bmatrix}}}\end{matrix}$

The same features of the equation as those given above may be omitted.Referring to FIG. 10 , the outer pixel data PD2′ may be estimated usingthe two nodes N1 and N2 nearest to the outer pixel data PD2′ and thenodes N4 and N5 nearest to the outer pixel data PD2′ in the verticaldirection. For example, nodes N1 and N2 are closer to the outer pixeldata PD2′ when compared to nodes N4 and N5.

According to an embodiment of the present inventive concept, the outerpixel data PD2′ may be estimated by averaging a result of performinglinear interpolation on the nodes N1 and N4 nearest to the outer pixeldata PD2′ in the vertical direction and a result of performing linearinterpolation on the nodes N2 and N5 nearest to the outer pixel dataPD2′ in the vertical direction.

According to an embodiment of the present inventive concept, when theinterpolation methods described with reference to FIGS. 9 and 10 areused, a loss of pixel information may be reduced. For example, besidesthe binning and interpolation methods described above, various binningmethods, e.g., weighted averaging and median-based binning, used in thefields of image processing and image analysis may be used, and variousinterpolation methods, e.g., nth order curve fitting and multivariateinterpolation, may be used. The binning and interpolation methodsdescribed above may be applied to various CFA patterns. Theinterpolation methods according to an embodiment of the presentinventive concept are not limited to the equations described withreference to FIGS. 8 to 10 .

Embodiments of the present inventive concept with reference to FIGS. 8to 10 may use the same interpolation method for central pixel data.Embodiments of the present inventive concept with reference to FIGS. 8to 10 may use different interpolation methods for corner pixel data andouter pixel data from one another. The interpolation method for cornerpixel data and the interpolation method for outer pixel data, which aredescribed with reference to each of FIGS. 8 to 10 , may be examples.During reconstruction from second pixel data, the interpolation methodaccording to an embodiment of the present inventive concept withreference to FIG. 8 may be used for corner pixel data, and theinterpolation method according to an embodiment of the present inventiveconcept with reference to FIG. 9 may be used for outer pixel data.

Although the embodiments of reconstructing N×N pixel data from(N−1)×(N−1) pixel data have been described, the present inventiveconcept is not limited thereto. According to an embodiment of thepresent inventive concept, N×N pixel data may be reconstructed from(N−2)×(N−2) pixel data by using the interpolation methods describedabove. According to an embodiment of the present inventive concept, N×Npixel data may be reconstructed from (N−1)×(N−2) pixel data.

FIGS. 11A and 11B are flowcharts of an image data processing methodaccording to an embodiment of the present inventive concept.

FIG. 11A is a flowchart of a method of processing image data output froma CFA including N×N same color pixels.

First pixel data in an N×N array, which is output from color pixels, maybe converted into second pixel data in an (N−L)×(N−M) array in operationS100, and third pixel data in the N×N array may be generated byperforming reconstruction on the second pixel data in operation S200.

FIG. 11B is a detailed flowchart of the image data processing method ofFIG. 11A.

The second pixel data may be obtained by forming (N−L)×(N−M) nodes byperforming a binning operation on the first pixel data in operationS110.

A piece of third pixel data may be generated by interpolating at leastone of the (N−L)×(N−M) nodes in operation S210.

According to an embodiment of the present inventive concept, the thirdpixel data in the N×N array may include at least one piece of centralpixel data and a plurality of pieces of edge pixel data surrounding thecentral pixel data. The edge pixel data may include outer pixel dataadjacent to the central pixel data in the X-axis direction or the Y-axisdirection and/or corner pixel data adjacent to the central pixel data ina diagonal direction. According to the present embodiment, the X-axisdirection may correspond to a row direction, and the Y-axis directionmay correspond to a column direction.

The central pixel data may be generated by interpolating four nodessurrounding the central pixel data in operation S310.

The outer pixel data may be generated by interpolating two nodesadjacent to the outer pixel data in operation S320. Alternatively, theouter pixel data may be generated by performing linear interpolation orinterpolation using a second-order polynomial on two nodes adjacent tothe outer pixel data and additional nodes in the same columns or row asthe two nodes in operation S320.

The corner pixel data may be generated by obtaining a value of one nodeadjacent to the corner pixel data in operation S330. Alternatively, thecorner pixel data may be generated by performing linear interpolation orinterpolation using a second-order polynomial on one node adjacent tothe corner pixel data and additional nodes in a diagonal direction inoperation S330.

When the image data processing method according to an embodiment of thepresent inventive concept is used, phase artifacts occurringperiodically or aperiodically may be removed. When a remosaic algorithmconverting a non-Bayer pattern into a Bayer pattern is used withoutusing the image data processing method according to an embodiment of thepresent inventive concept, artifacts may be emphasized as a certainpattern

However, when the image data processing method according to anembodiment of the present inventive concept is used, pattern noise maybe removed. Because adjacent pixel information is not used in afrequency domain but is used in a spatial domain in the image dataprocessing method according to an embodiment of the present inventiveconcept, the image data processing method may be used, regardless ofexternal factors, such as heat and color temperature, or intrinsiccharacteristics of various CFA patterns and sensor modules, and separatecalibration might not be necessary.

While the present inventive concept has been described with reference toembodiments thereof, it will be understood by those of ordinary skill inthe art that various changes in form and details may be made theretowithout departing from the spirit and scope of the present inventiveconcept.

What is claimed is:
 1. A method of processing image data, the methodcomprising: receiving image data from a color filter array including N×Nsame color pixels; converting first pixel data in an N×N array intosecond pixel data in an (N−L)×(N−M) array, wherein the first pixel datais output from the N×N same color pixels; and generating third pixeldata in the N×N array by performing reconstruction on the second pixeldata, wherein each of “L” and “M” is a natural number that is greaterthan or equal to 1 and less than N, and “N” is a natural number that isgreater than or equal to
 2. 2. The method of claim 1, wherein theconverting of the first pixel data into the second pixel data includesforming (N−L)/(N−M) nodes through binning, and each of the (N−L)×(N−M)nodes includes data information of pixels of the first pixel data,wherein each of the (N−L)×(N−M) nodes are adjacent to the pixels.
 3. Themethod of claim 2, wherein the generating of the third pixel dataincludes generating a piece of the third pixel data by interpolating atleast two nodes among the (N−L)×(N−M) nodes.
 4. The method of claim 3,wherein the third pixel data in the N×N array includes at least onepiece of central pixel data and a plurality of pieces of edge pixel datasurrounding the at least one piece of central pixel data.
 5. The methodof claim 4, wherein the at least one piece of central pixel data isgenerated by interpolating four nodes surrounding the at least one pieceof central pixel data.
 6. The method of claim 4, wherein the pluralityof pieces of edge pixel data include: a plurality of pieces of outerpixel data adjacent to the at least one piece of central pixel data inan X-axis direction or a Y-axis direction, which are perpendicular toeach other; and a plurality of pieces of corner pixel data adjacent tothe at least one piece of central pixel data in a diagonal directionwith respect to the X-axis direction and the Y-axis direction.
 7. Themethod of claim 6, wherein each of the plurality of pieces of outerpixel data is generated by interpolating two nodes adjacent thereto. 8.The method of claim 6, wherein each of the plurality of pieces of outerpixel data is generated by performing linear interpolation orinterpolation using a second-order polynomial on two nodes adjacentthereto and additional nodes in same columns or rows as the two nodes.9. The method of claim 6, wherein each of the plurality of pieces ofcorner pixel data is generated by performing linear interpolation orinterpolation using a second-order polynomial on one node adjacentthereto and additional nodes in a diagonal direction with respect to theX-axis direction and the Y-axis direction.
 10. A method of processingimage data, the method comprising: receiving image data output from acolor filter array (CFA) including a plurality of CFA blocks includingat least one color region including pixels of a same color; generatingsecond pixel data by converting first pixel data output from the atleast one color region; and generating third pixel data by performingreconstruction on the second pixel data, wherein the generating of thesecond pixel data includes forming a plurality of nodes includinginformation about pixels included in the first pixel data.
 11. Themethod of claim 10, wherein the forming of the plurality of nodesincludes setting a conversion unit and performing binning with a centerof the conversion unit being set as each of the plurality of nodes. 12.The method of claim 11, wherein each of the plurality of nodescorresponds to a contact point between a plurality of pieces of pixeldata included in the third pixel data.
 13. The method of claim 12,wherein the third pixel data includes: central pixel data surrounded byfour nodes among the plurality of nodes; outer pixel data in contactwith two nodes among the plurality of nodes; and corner pixel data incontact with one node among the plurality of nodes.
 14. The method ofclaim 13, wherein the central pixel data is generated by interpolatingthe four nodes surrounding the central pixel data.
 15. The method ofclaim 13, wherein the outer pixel data is generated by interpolationusing an average value of the two nodes adjacent with the outer pixeldata or generated by performing linear interpolation or interpolationusing a second-order polynomial on the two nodes adjacent with the outerpixel data and additional nodes in a same columns as the two nodes. 16.The method of claim 13, wherein the corner pixel data is generated byobtaining a value of the one node adjacent to the corner pixel data orby performing linear interpolation or interpolation using a second-orderpolynomial on the one node adjacent to the corner pixel data andadditional nodes in a diagonal direction with respect to an X-axisdirection and a Y-axis direction, which are perpendicular to each other.17. The method of claim 10, wherein a method of performing thereconstruction on the second pixel data varies with a number of nodescontacting a piece of pixel data included in the third pixel data. 18.An image processing processor processing image data output from an imagesensor, the image processing processor comprising: a first processingcircuit configured to perform conversion of the image data received fromthe image sensor; and a second processing circuit configured to performreconstruction on converted data, wherein the first processing circuitis configured to perform the conversion by binning the image dataincluding pixel data in an N×N array into pixel data in an (N−L)×(N−M)array, where each of “L” and “M” is a natural number that is greaterthan or equal to 1 and less than “N”, “N” is a natural number that isgreater than or equal to 2, and the second processing circuit isconfigured to perform the reconstruction by interpolating nodescorresponding to the pixel data in the (N−L)×(N−M) array.
 19. The imageprocessing processor of claim 18, wherein the second processing circuitis configured to reconstruct pixel data surrounded by four nodes amongthe nodes corresponding to the pixel data in the (N−L)×(N−M) array byperforming interpolation using an average value of the four nodes. 20.The image processing processor of claim 18, wherein the secondprocessing circuit is configured to reconstruct pixel data contactingone or two nodes among the nodes corresponding to the pixel data in the(N−L)×(N−M) array by performing linear interpolation or interpolationusing a second-order polynomial on the one or two nodes.